feat(ai-research): keep v4 url history per team month - #117983

🤖 CI report

[Medium impact] Changes image fetch history expiry logic for a new URL format.

The missing-result convention must be fixed before merging; the cache test gap is non-blocking.

_{Reviews (1) · Last reviewed commit: "feat(ml-image-fetch): keep v4 url histor..." · Reviewed by Greptile}

Risk: No findings

This increment hardens the v4 crawl-history store: malformed DynamoDB rows now read as cache misses with a warning instead of failing the batch, legacy rows carrying a plaintext response URL are stripped on read so their validators read as unbound, the cache field becomes a keyed hash (responseUrlHmac), URL-history entries now expire 8 days after their team month instead of with the month key row, and fetched zstd bodies must be exactly one complete frame. All behavior changes fail toward refetching rather than skipping, and no exploitable path was found.

_{Sentinel reviewed 05760b0 · Review settings}

Written by Claude on Robbie's behalf, from a QA review of the v4 stack. These are minor findings: none of them blocks the merge.

• One malformed history item crash-loops the lane and makes rollback unsafe. parseStoredItem in dynamodb-crawl-history.ts throws on any item that fails its checks. The throw fails read(), the batch and the pod, and the replay reads the same item again. A newer build that writes a new kind or status would also crash-loop an older build after a rollback. Suggestion: treat an item that does not parse as a cache miss, with a log line and a counter.

Written by Claude on Robbie's behalf, from a QA review of the v4 stack.

The fetch history keeps the full image URL in plaintext for months.

• cacheMetadata in image-fetcher.ts sets responseUrl to the requested URL. FetchRunner.terminal writes the whole cache object as JSON into ai-research-v4-image-fetch-history, under the key imageurl:v4:{team}:{month}:{hash}. So each row links a team to a raw image URL, query string included.
• The rows have a TTL, but it is the team month key expiry (urlHistoryStorageExpiresAtMs returns teamMonthKeyExpiresAtMs): the start of the fifth month after the key month. That is 4 to 5 months, against 38 days for the images themselves.
• A shred does not reach this table, so the URL stays readable after its key is gone. The infra README says the opposite: "a shredded month key makes them unlinkable to a URL".
• v3 does neither. Its HttpCacheMetadata has no responseUrl, and its history for a month-scoped ref expires 8 days after the end of that month.

responseUrl is read in one place: the equality check that decides whether to send validators (previousCache?.responseUrl === url). Suggestions:

1. Store an HMAC of the URL (the lane already holds the team month key) or a SHA-256 of it, and compare that. Or drop the field and send no validators after a redirect.
2. Use v3's expiry for v4 refs: the end of the ref's month plus 8 days. A ref carries the month of its write, so jobs for it stop when that month ends, and nothing reads the item after that.

With both, the table holds no URL, and each item goes away a few weeks after its month. Fixing this before the first sync avoids a table scan to clean up rows already written.

This pull request was merged into master as part of stacked PR 119750.

Deploy status